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ABSTRACT 

An evaluation was conducted to determine the outcomes 
and impacts of participation in the ALBERTA BEST training program in 
terms of participant attitudes toward service excellence and business 
profitability. ALBERTA BEST is a customer service program offered by 
the Alberta (Canada) government. The evaluation involved a series of 
case studies conducted in a sample of tourism businesses during a 
3-month period. This paper was prepared as a case study itself to 
explain how the theory was developed, cases were selected, measures 
were defined, and data were collected and aggregated into nine 
separate but similar case studies. A work plan presented in January 
of 1992 eventually developed into a statement of eight evaluation 
objectives. Case study sites were selected in three segments of the 
tourism industry. Sites were divided according to whether 60 percent 
of employees had received ALBERTA BEST training. Further definition 
of study objectives preceded data collection and instrument 
development. From February 7 through March 5, 1992, the study sites 
were visited. Data were aggregated through a developed data key to 
code and sort survey data, interview data, financial data, document 
analysis, and on-site observations. The final report was produced in 
April from the compendium. One figure illustrates the evaluation 
design. (SLD) 
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The purpose of the evaluation study reported here, entitled, "Impact of Service Excellence 
and ALBERTA BEST Training on a Tourism Business: Case Studies," was to determine the 
outcomes and impacts of participation in the ALBERTA BEST training program in terms 
of both participant attitudes towards service excellence and business profitability. It was the 
view of the client, Alberta Tourism, Parks and Recreation, that the impacts of the training 
program would have to be longitudinal in nature. As the program had only been in 
operation for one year, this evaluation was seen as a baseline for future research activities. 
At the same time, the client had a strong need for anecdotal evidence related to program 
impact and thus it was pre-determined that study methodology would be employed. 

A case study has been defined by Yin (1988:23) as an empirical inquiry that: 

1. Investigates a contemporary phenomenon within its real-life context; 

2. The boundaries between phenomenon and context are not clearly evident; and 
in which 

3. Multiple sources of evidence are used. 

Yin also provides a model for a replication approach to multiple case studies (1988:56) and 
describes the process as consisting of theory development, case selection, definition of 
specific measures and the data collection process (1988:57). Each individual case study 
consists of a **whole" study in which "... convergent evidence is sought regarding the facts 
and conclusions for the case; each case's conclusions are then considered to be the 
information needing replication by other individual cases." Each case should show how and 
why a particular proportion was or was not demonstrated and the cross-case analysis should 
indicate why certain cases were predicted to have certain results, vis a vis the proposition, 
why others were not and what the actual findings were. 

In the spirit of case study research, this paper was prepared as a case study itself to explain 
how the theory was developed, and cases were selected, how measures were defined, and 
how data was collected and aggregated into nine separate but similar case studies. 



L Theory Development and Case Selection 

LI The Request for Proposals (December 13, 1991) 

The study in question was an evaluation of a customer service program offered by 
the Alberta Government entitled ALBERTA BEST. In general terms the study was 
to involve a series of case studies to be conducted in a selection of tourism 
businesses during a three-month period. The objectives outlined in the Request for 
Proposals (RFP) were fairly brief: 




Purpose of the Study 

1. To determine from a business's perspective the impacts of ALBERTA BEST 
training with respect to achieving service excellence. 

2. To gather information from participants of ALBERTA BEST seminars (e.g., 
testimonials). This information would be used in the awareness component 
and in promoting ALBERTA BEST seminars. 

3. To acquire information on the effects of managing for sendee excellence to a 
business's profitability. This information would be used in motivating and 
educating stakeholders in individual tourism businesses on the need to 
understand and to pursue service excellence. This would ultimately promote 
the ALBERTA BEST seminars. 

Terms of Reference 

Impacts of Service Excellence and ALBERTA BEST Training 

on a Tourism Business 



While at first glance, these seemed like simple objectives, it quickly became apparent 
that there was a definite positive bias built into study expectations. The challenge was 
to not compromise my ethics as an evaluator while at the same time providing the 
client withe the information required. 



L2- The Proposal (December 18, 1991) 

There is a trick to writing objectives in a proposal. On the one hand you want to 
make it clear that you can offer what it is the client desires. On the other hand, you 
want to give the objectives your own personal stamp and be persuasive enough to 
convince them that you are the only ones who can really meet their information 
needs. 

In this case, my concern over the perceived slant of desired outcomes vas not 
lessened by a phone call to a staff member at the Department who ulithely 
responded to my cautious queries, "We want everything!" 

As a result, my tactic was to return vague desires with vague responses. I replied with 
a series of questions, rather than objectives, based on Brinkerhoffs six-stage training 
evaluation model (1987) with particular reference to Stage 5, Using Learning on the 
Job and Stage 6, How the Training Investment Paid Off. 
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1. What are the impacts of ALBERTA BEST training on employees, managers, 
customers and owners? 

2. Have the components of service excellence (i.e., service strategy, service 
culture and service support systems) been implemented, and if so, to what 
extent? 

3. What changes have resulted in organizations which have participated in 
ALBERTA BEST training in terms of profitability, morale, turnover and 
productivity? 

4. What barriers exist to either program participation or the implementation of 
program principles in the industry? 

Proposal to Evaluate 

The Impacts of Service Excellence and ALBERTA BEST Training 

On a Tourism Business 
Gail V. Barrington & Associates 



In preparing these questions, I separated out the concepts of service excellence and 
ALBERTA BEST training and added the items on change and barriers, based on a 
close reading of the RFP and a "best guess" about what was really required. 

L3 The Work Plan (January 10, 1992) 

Having been awarded the contract on January 2, 1992, we met with the client to 
review the proposal in order to clarify any areas in need of revision. This initial client 
meeting is critical to the entire evaluation process yet as the evaluator you are at 
your weakest point because you have the least information you will ever have about 
the program. You will have to live with the decisions made now, and likely as not, 
they will come back to haunt you. 

My main objective at this initial meeting was to clarify my ethical position; namely, 
that I could not promise that the evaluation would produce positive results, nor 
could I guarantee that the report could be used for promotional purposes. I received 
a qualified response. It also became patently clear that I really had two clients: one 
manager v/ho wanted an evaluation of the ALBERTA BEST program; and another 
manager wanted a reading on service excellence in the province, whether the 
ALBERTA BEST program was involved or not. 

During the following week I read some background material, consulted with my 
content expert reviewed the meeting notes and developed the Work Plan. 
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At this point, to clarify what I was to do even further, I used an evaluation model 
as a template and laid it over the proposed study objectives to determine if there 
were any gaps or overlaps. I used the model developed by my office in 1990 which 
we have used successfully in a number of evaluations, but most likely any number of 
models could have served the same purpose. Our model looked as follows: 



Figure 1 Evaluation Design Model 
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The result of this process was that the four initial objectives became eight: 



1. To determine what the impacts of ALBERTA BEST training are on employees, 
managers/owners and customers in selected case study sites. 

2. To determine if the components of service excellence (i.e., service strategy, 
service culture and service support systems) have been implemented in case 
study sites, and if so, to measure the extent of implementation and outcomes. 

3. To determine what changes have resulted in case study organizations which 
have participated in ALBERTA BEST training in terms of profitability, morale, 
turnover and productivity and to assess the trend, 

4. To determine if barriers exist to either program participation or the 
implementation of program principles in cas. study sites and if so, to identify 
those barriers. 

5. To determine if any unanticipated outcomes have resulted from case study 
sites' participation in ALBERTA BEST training programs, 

6. To determine if there are any unresolved issues related to the involvement of 
case study sites in their participation in ALBERTA BEST training programs. 

7. To develop an internal and external tracking system to measure the impacts of 
ALBERTA BEST, based on the experience of this pilot study, for future use by 
Alberta Tourism, Parks and Recreation and organizations involved in the 
ALBERTA BEST program, 

8. To present the findings of this evaluation in a final report. 

Work Plan 

Impacts of Service Excellence and ALBERTA BEST Training 

on a Tourism Business 
Gail V, Harrington & Associates 



During this period, the number of case studies required by the client had increased 
from six to nine, with a commensurate increase in dollars but not time. I was anxious 
to nail down the study and so all the objective:: were limited by the words, "at case 
study sites," The extent of service excellence to b2 measured was expanded to 
include outcomes as well as implementation. Objective #3 was expanded to include 
an assessment of the trend in changes to profitability, morale, turnover and 
productivity at the request of the client. Four objectives were added, two as a result 
of the model (unanticipated outcomes and unresolved issues), the one regarding the 
tracking system which was seen by the client as a logical extension of our study, and 
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the last one regarding the final report which I added on a hopeful note to ensure 
that closure was at hand. 



Also during this period the case study site were also selected, A matrix was 
developed which categorized sites according to whether or not 60% of employees 
had received ALBERTA BEST training and whether or not management had made 
a commitment to service excellence. 







ALBERTA BEST 
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Training 




Training 
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Hotel 
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Food & Beverage 


#4 


Hotel 


Components 
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Outdoor 
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Food & Beverage 






Recreation 






Few Service Excellence 


#6 


Hotel 


#8 


Hotel 


Components 


#7 


Food & Beverage 


#9 


Outdoor Recreation 



Three sectors of the tourism industry were selected including hotels, food and 
beverage and outdoor recreation. Location and size were also considered. Where 
possible, a random selection was made based on various industry lists; in a few cases, 
selection was made based on an industry expert's best guess. 

The Data Collection Matrix was attached to the Work Plan and was similar in nature 
to the one we had used successfully in a recent study. It outlined Research Areas 
and Methods in a simple fashion and applied a numbering system to them. (Consult 
Figure 1). 

2, Definition of Measures 

2.1 Instrument Development (January 10-30, 1992) 

Based on this outline, we forged ahead, bearing in mind our three month deadline, 
and we developed a series of five instruments to be administered at the nine case 
study sites. These were as follows: 

1. Employee Survey 

2. Supervisor Survey 

3. Owner/Manager Survey 

4. Customer Survey 

5. Records Analysis Checklist 
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And this is where the study began to founder. We fell into a morass of ambiguous 
terms including service culture, service strategies, service recovery, and service 
excellence. Which was which? Which ones were involved in ALBERTA BEST 
training, and which ones were not? And how could you tell? 

Not surprisingly, the instalments evoked copious pages of finely written comments, 
suggestions, and revisions from the client Their suggestions moved farther and 
farther from the direction in which I had thought I had been headed. As I tried to 
incorporate the changes, it quickly became apparent that I did not know where we 
were headed. I called a halt 



2.2 Data Collection Matrix #2 (February 1 - 6, 1992) 

By this time we had models, objectives, criteria, guidelines, factors, components and 
survey items coming out of our ears. So we went back to the drawing board and 
asked ourselves what we were really working with. Our answer was the objectives and 
research topics which had been defined in the Work Plan, the ALBERTA BEST 
course outline and the components of service excellence as initially outlined in the 
RFP. 

It was obvious that we have not distinguished clearly enough between 
implementation and outcomes in our instruments. The topics on implementation in 
the original Data Collection Matrix, policy, management, staff and supplies, were far 
too brief to reflect what was covered in the training program which had a real focus 
on systems required to ensure service excellence as an outcome. 

Our first decision was that not every instrument would cover every topic; that some 
people were more qualified to comment on certain areas than others. The result of 
this discussion was our Concept Map. (Consult Figure 2). 

Then we aligned the objectives with the research topics and the more specific 
research ajeas, developed an expanded Data Collection Matrix. We added a whole 
section on implementation systems which resulted in a sixth instrument entitled, 
"Service Excellence Systems Checklist" Then we cut and pasted survey items to fit 
Those that did not fit hit the cutting room floor. Where there were gaps, we added 
questions. The patchwork quilt which resulted covered the top of our board room 
table but we were finally pleased with the result (Consult Figure 3 for an excerpt 
from our expanded Data Collection Matrix.) The surveys were then pieced back 
together. They were cross referenced with the appropriate Data Collection Matrix 
code and the Data Collection Matrix was coded with the appropriate survey item. 
(Consult Figure 4 for an example from our Employee Survey.) 

We sent the revised surveys to the client along with the Conceptual Map and the 
Expanded Data Collection Matrix and said, "This is what we are doing and this is 
all we are doing." There were no more suggestions forthcoming from the client as 
the staff members in both departments were satisfied that their information needs 
would be met 
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It was during this short period that our evaluation moved from being a faihire in the 
making to being a success. In the redefinition of the purpose of the various 
instruments and the development of a comprehensive aggregation tool (the Data 
Collection Matrix) the study was placed firmly on track and it rolled out from there 
in a predictable fashion. 

Data Collection and Aggregation 

3.1 Case Study Research (February 7 - March 5, 1992) 

During this period the nine case study sites were visited by our two teams, all six 
instruments were administered at each site, documents were reviewed and general 
observations made. 



3.2. The Data Keys (March 5 -31, 1992) 

Further to the Data Collection Matrix, we developed the Data Key as a way to 
aggregate data from all the different sources on each topic in the Data Collection 
Matrix. All survey data, interview data, financial data, document analysis, and on-site 
observations were coded and sorted according to the Data Collection Matrix codes. 
The result was a compendium from which the final report could be produced, 
(Consult Figure 5 for an excerpt from one of our data keys on the topic of employee 
orientation to seivice strategy at Billy Joe's Restaurant). 



3.3 The Final Report (April 1 - 30, 1992) 

In the end, it was straightforward work putting the final report together. Our only 
problem was its length as we had to run through the Data Collection Matrix for each 
of the nine case studies. (Consult Figure 6 for the excerpt from our final report 
which describes study findings on employee orientation at Billy Joe's Restaurant.) 
Even the cross-case analysis was straightforward once the process had been clarified. 
The feedback from the client was very favourable, but perhaps, more importantly, 
as researchers we felt that we had slain a dragon of immense proportions by the use 
of these aggregation tools. 
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